Scheduling and Energy Efficiency Improvement Techniques for Hadoop Map-reduce: State of Art and Directions for Future Research

MapReduce has become ubiquitous for processing large data volume jobs. As the number and variety of jobs to be executed across heterogeneous clusters are increasing, so is the complexity of scheduling them efficiently to meet required objectives of performance. This report presents a survey of some of the MapReduce scheduling algorithms proposed for such complex scenarios. A taxonomy is provided for Map-reduce algorithms based on their runtime nature. The algorithms proposed for each hierarchical level of MapReduce scheduling are described in detail. Some pointers for future research to further improve the scheduling techniques are provided. Another aspect of MapReduce is that the size of their clusters is usually in hundreds and thousands, while it is used for processing infrequent batch and interactive jobs in parallel across these machines. Thus there is a need to look at energy efficiency of MapReduce clusters. A survey of some of the techniques proposed to improve MapReduce energy efficiency is done. The studied techniques have been classified based upon the MapReduce component they work-on. Details of techniques in each category are provided. Few suggestions for future research are given based on the gaps observed in these works.